feat(ai): Add $ai_framework property for framework integrations
#347
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Adds framework identification metadata to all AI events for easier filtering and analytics. Each integration now includes a $ai_lib_metadata property with schema version and framework name. - LangChain: Hardcoded to "langchain" - Native wrappers (Anthropic, OpenAI, Gemini): Uses provider name - Ready for future frameworks (pydantic-ai, crewai, llamaindex) This enables PostHog queries to easily distinguish between: - Direct SDK wrapper usage - Framework-mediated usage (LangChain, etc.) - Different framework types 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Added \$ai_lib_metadata to call_llm_and_track_usage (sync) - Added \$ai_lib_metadata to call_llm_and_track_usage_async (async) - Added test assertion in test_basic_completion - Placed metadata at end of properties for consistency All tests pass successfully. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
andrewm4894
changed the title
$ai_lib_metadata to AI integrations
Creates a single `get_ai_lib_metadata(framework)` utility function to generate the $ai_lib_metadata object, replacing inline implementations across the codebase. Changes: - Add get_ai_lib_metadata() utility to utils.py - Update LangChain callbacks to use utility function - Update call_llm_and_track_usage() to use utility function - Update call_llm_and_track_usage_async() to use utility function - Update capture_streaming_event() to use utility function Benefits: - Consistency across all integrations - Single source of truth for metadata structure - Easier to extend with version detection later 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
482
to
491
|
|
||
| assert call_args["distinct_id"] == "test-id" | ||
| assert call_args["event"] == "$ai_generation" | ||
| assert props["$ai_lib_metadata"] == { | ||
| "schema": "v1", | ||
| "frameworks": [{"name": "openai"}], | ||
| } | ||
| assert props["$ai_provider"] == "openai" | ||
| assert props["$ai_model"] == "gpt-4" | ||
| assert props["$ai_input"] == [{"role": "user", "content": "Hello"}] |
There was a problem hiding this comment.
style: Check that Anthropic, Gemini, and LangChain tests also assert $ai_lib_metadata
This test correctly validates the new metadata field, but similar assertions are missing in other provider tests:
posthog/test/ai/anthropic/test_anthropic.py:276(test_basic_completion)posthog/test/ai/gemini/test_gemini.py:147(test_new_client_basic_generation)posthog/test/ai/langchain/test_callbacks.py:160(test_basic_chat_chain)
Prompt To Fix With AI
This is a comment left during a code review.
Path: posthog/test/ai/openai/test_openai.py
Line: 482:491
Comment:
**style:** Check that Anthropic, Gemini, and LangChain tests also assert `$ai_lib_metadata`
This test correctly validates the new metadata field, but similar assertions are missing in other provider tests:
- `posthog/test/ai/anthropic/test_anthropic.py:276` (`test_basic_completion`)
- `posthog/test/ai/gemini/test_gemini.py:147` (`test_new_client_basic_generation`)
- `posthog/test/ai/langchain/test_callbacks.py:160` (`test_basic_chat_chain`)
How can I resolve this? If you propose a fix, please make it concise.Add missing $ai_lib_metadata assertions to Anthropic, Gemini, and LangChain tests to match the validation already present in OpenAI tests. Each test now verifies the metadata field contains the correct schema version and framework name. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Radu-Raicea
reviewed
Oct 24, 2025
Changes:
- Replace complex $ai_lib_metadata object with simple $ai_framework string
- Only include $ai_framework when using actual framework (LangChain)
- Remove $ai_framework from direct provider calls (OpenAI, Anthropic, Gemini)
- Update all tests to reflect new behavior
Before: {"schema": "v1", "frameworks": [{"name": "langchain"}]}
After: "langchain" (only when using LangChain framework)
This eliminates wasteful redundancy where framework=provider for direct calls.
andrewm4894
changed the title
$ai_lib_metadata to AI integrations
andrewm4894
changed the title
$ai_framework property for framework integrations
andrewm4894
added a commit
that referenced
this pull request
Oct 28, 2025
Update version and changelog for PR #347
andrewm4894
added a commit
that referenced
this pull request
Oct 28, 2025
Update version and changelog for PR #347
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Adds a simple
$ai_frameworkproperty to AI events, only when an actual framework layer is used (e.g., LangChain). This helps identify which abstraction layer is being used on top of the underlying AI provider.Changes
$ai_frameworkproperty to AI events for framework integrations (LangChain)"langchain")Example Events
Framework call (LangChain):
{ "$ai_framework": "langchain", "$ai_provider": "openai", "$ai_model": "gpt-4", ... }Direct provider call (OpenAI):
{ "$ai_provider": "openai", "$ai_model": "gpt-4", ... }Rationale
This approach eliminates redundancy - we only add
$ai_frameworkwhen there's an actual framework layer. For direct provider calls, the framework would just duplicate the provider name, which is wasteful.Testing
$ai_frameworkin LangChain tests